Performance boost of time-delay reservoir computing by non-resonant clock cycle

The time-delay-based reservoir computing setup has seen tremendous success in both experiment and simulation. It allows for the construction of large neuromorphic computing systems with only few components. However, until now the interplay of the different timescales has not been investigated thoroughly. In this manuscript, we investigate the effects of a mismatch between the time-delay and the clock cycle for a general model. Typically, these two time scales are considered to be equal. Here we show that the case of equal or resonant time-delay and clock cycle could be actively detrimental and leads to an increase of the approximation error of the reservoir. In particular, we can show that non-resonant ratios of these time scales have maximal memory capacities. We achieve this by translating the periodically driven delay-dynamical system into an equivalent network. Networks that originate from a system with resonant delay-times and clock cycles fail to utilize all of their degrees of freedom, which causes the degradation of their performance

Ground-state modulation-enhancement by two-state lasing in quantum-dot laser devices

We predict a significant increase of the 3 dB-cutoff-frequency on the ground- state lasing wavelength for two-state-lasing quantum-dot lasers using a microscopically motivated multi-level rate-equation model. After the onset of the second lasing line, the excited state acts as a high-pass filter, improving the ground-state response to faster modulation frequencies. We present both numerically simulated small-signal and large-signal modulation results and compare the performance of single and two-state lasing devices. Furthermore, we give dynamical arguments for the advantages of two-state lasing on data-transmission capabilities

Multiplexed networks: reservoir computing with virtual and real nodes

The reservoir computing scheme is a machine learning mechanism which utilizes the naturally occurring computational capabilities of dynamical systems. One important subset of systems that has proven powerful both in experiments and theory are delay-systems. In this work, we investigate the reservoir computing performance of hybrid network-delay systems systematically by evaluating the NARMA10 and the Sante Fe task for varying system parameters. We construct ‘multiplexed networks’ that can be seen as intermediate steps on the scale from classical networks to the ‘virtual networks’ of delay systems. We find that the delay approach can be extended to the network case without loss of computational power, enabling the construction of faster reservoir computing systems.DFG, 163436311, SFB 910: Kontrolle selbstorganisierender nichtlinearer Systeme: Theoretische Methoden und Anwendungskonzept

Information-theoretical analysis of statistical measures for multiscale dynamics

Multiscale entropy (MSE) has been widely used to examine nonlinear systems involving multiple time scales, such as biological and economic systems. Conversely, Allan variance has been used to evaluate the stability of oscillators, such as clocks and lasers, ranging from short to long time scales. Although these two statistical measures were developed independently for different purposes in different fields in the literature, their interest is to examine multiscale temporal structures of physical phenomena under study. We show that, from an information-theoretical perspective, they share some foundations and exhibit similar tendencies. We experimentally confirmed that similar properties of the MSE and Allan variance can be observed in low-frequency fluctuations (LFF) in chaotic lasers and physiological heartbeat data. Furthermore, we calculated the condition under which this consistency between the MSE and Allan variance exists, which is related to certain conditional probabilities. Heuristically, physical systems in nature including the aforementioned LFF and heartbeat data mostly satisfy this condition, and hence the MSE and Allan variance demonstrate similar properties. As a counterexample, an artificially constructed random sequence is demonstrated, for which the MSE and Allan variance exhibit different trends

Deep neural networks using a single neuron: folded-in-time architecture using feedback-modulated delay loops

Deep neural networks are among the most widely applied machine learning tools showing outstanding performance in a broad range of tasks. We present a method for folding a deep neural network of arbitrary size into a single neuron with multiple time-delayed feedback loops. This single-neuron deep neural network comprises only a single nonlinearity and appropriately adjusted modulations of the feedback signals. The network states emerge in time as a temporal unfolding of the neuron’s dynamics. By adjusting the feedback-modulation within the loops, we adapt the network’s connection weights. These connection weights are determined via a back-propagation algorithm, where both the delay-induced and local network connections must be taken into account. Our approach can fully represent standard Deep Neural Networks (DNN), encompasses sparse DNNs, and extends the DNN concept toward dynamical systems implementations. The new method, which we call Folded-in-time DNN (Fit-DNN), exhibits promising performance in a set of benchmark tasks. Development of deep neural networks benefits from new approaches and perspectives. Stelzer et al. propose to fold a deep neural network of arbitrary size into a single neuron with multiple time-delayed feedback loops which is also of relevance for new hardware implementations and applications.DFG, 183049896, GRK 1740: Dynamische Phänomene in komplexen Netzwerken: Grundlagen und AnwendungenDFG, 411803875, Dynamik gekoppelter Systeme mit Zeitverzögerungen und deren AnwendungenEC/H2020/952060/EU/Transparent, Reliable and Unbiased Smart Tool for AI/TRUST-AITU Berlin, Open-Access-Mittel – 202

Symmetriebrechende Bifurkationen und Reservoir-Computing in regulären Oszillatornetzwerken

The focus of this thesis is the investigation of regular oscillator networks with numerical and analytical tools. Such systems are ubiquitous in nature and can emerge in the shape of vastly different processes. Among others, they can be mathematically derived as discrete approximations of continuous wave equations or as a local approximation of coupled dynamical systems. In the most basic meaning of the word, an oscillator is a system that shows a periodic change in one of its state variables. This thesis will focus on two different mathematical models: The Stuart-Landau oscillator that is popular in bifurcation theory and the more applied Lang-Kobayashi model for laser dynamics. Lasers can be seen as optical oscillators and they are most accessible of all oscillatory systems for precise and fast experiments. Many previous works have covered different aspects of coupled oscillators, with some effects such as synchronization having been studied in some form for more than three centuries. The overall literature on the topic of coupling-induced dynamics is vast, and many different subtopics can be identified, such as the influence of delay, the application of control schemes, the properties of biological and artificial neural networks, the causes of chaotic motion or the importance of stochastic processes. The focus in this thesis will be on highly symmetric networks of coupled oscillators, where the local dynamics follow very simple models. The focus is therefore on the coupling-induced effects in general and not on elaborate and extremely detailed descriptions of specific experiments. However, because the effects of coupling and especially those connected to symmetries are in some sense universal, generic models for oscillators and lasers more than suffice for the investigations done here. A particular focus will be on the emergence of symmetry-broken states in rings of oscillators. Several different novel aspects of these symmetry-breaking mstates are discussed and their connections to the established solutions from the literature are explored. The second part of this thesis concerns itself with one of the possible uses of such oscillatory and laser networks: The neuro-inspired machine-learning concept called ’reservoir computing’. This neuromorphic computing approach allows the exploitation of the intrinsic complex behaviour of driven dynamical systems for analogue computing. The usability of regular networks of oscillators will be explored. Furthermore, as reservoir computing still suffers from a lack of quantitative theory, a few fundamental aspects of reservoir computing will also be explored with the help of the examples in this thesis. As will be shown, hybrid delay-network systems can be created that are very suitable as reservoir computers.Diese Doktorarbeit beschäftigt sich mit den dynamischen Eigenschaften von regulären Netzwerken nichtlinearer Oszillatoren. Diese Systeme sind paradigmatisch für eine große Klasse von Effekten und erscheinen in verschiedensten Formen in der Natur. Unter anderem können Netzwerke von Oszillatoren als Näherung für Wellengleichungen in linearen und nichtlinearen Medien hergeleitet und viele gekoppelte komplexe System durch gekoppelte nichtlineare Oszillatoren angenähert werden. In der simpelsten und weitesten Definition ist ein Oszillator schlicht ein physikalisches oder mathematisches Modell, dessen wesentliche Eigenschaft die periodische Änderung mindestens einer ihrer Größen ist. In dieser Arbeit werden zwei Arten von oszillierenden Systemen untersucht werden: Zum einen das mathematische, abstrakte Modell des Stuart-Landau-Oszillators, zum anderen das physikalisch komplexere Lank-Kobayashi-Modell für Laser. Laser können als optische Oszillatoren gesehen werden und sind daher eines der besten Systeme zur experimentellen Erforschung der Dynamik nichtlinearer Oszillatoren. Die wissenschaftliche Literatur zu gekoppelter Oszillatoren kann in Teilen auf eine lange Geschichte zurückgreifen. Synchronisierung wurde beispielsweise das erste mal bereits vor mehr als drei Jahrhunderten beschrieben. Daher ist die Zahl der wissenschaftlichen Arbeiten zu gekoppelten Oszillatoren und oszillierenden System sehr groß und beschäftigt sich mit vielen verschiedenen Themen, wie zum Beispiel dem Einfluss von Verzögerungen, Kontrollansätzen, biologischen und künstlichen neuronalen Netzwerken, den Ursachen deterministischen Chaos oder den Eigenschaften stochastischer Prozesse. Das Themefeld dieser Arbeit wird sich daher auf Netzwerke von hoher Symmetrie beschränken, wobei zusätzlich die lokale Dynamik möglichst simpel gehalten ist. Der Fokus liegt auf den durch die Kopplung verursachten Effekten, wodurch simple Modelle genügen. Da Symmetrie aber ein allgemeines Konzept der physikalischen Wissenschaften ist, können auch bereits in simplen Modellen viele relevante Effekte untersucht werden. Ein großer Fokus dieser Arbeit ist das Entstehen von symmetriegebrochenen Lösungen in Ringen von Stuart-Landau Oszillatoren. Verschiedene neue Eigenschaften dieser symmetriegebrochenen Zustände werden erörtert und in Bezug zu den bereits aus der bisherigen wissenschaftlichen Literatur bekannten Lösungen gesetzt. Der zweite Teil dieser Arbeit untersucht eine der möglichen Anwendungen solcher Ringnetzwerke mit nichtlinearen Oszillatoren. Das neurologisch inspirierte ‘Machine Learning’-Konzept unter dem Namen ‘Reservoir Computing’ erlaubt die Ausnutzung der intrinsischen Rechenkraft dynamischer Systeme als analoge Computer. Die Nutzbarkeit von regulären Netzwerken bestehend aus Stuart-Landau- Oszillatoren wird untersucht. Das Gebiet des ‘Reservoir Computings’ besitzt noch keine ausgereifte quantative Theorie, weswegen ein wesentlicher Teil dieser Arbeit die Erforschung einiger grundlegender Aspekte ist. Desweiteren wird ein neues System von Netzwerken mit verzögerter Kopplung untersucht und es wird gezeigt, dass diese sich sehr gut als analoge Computer eignen.DFG, SFB 910, Kontrolle selbstorganisierender nichtlinearer Systeme: Theoretische Methoden und Anwendungskonzept

Model-free inference of unseen attractors: Reconstructing phase space features from a single noisy trajectory using reservoir computing

Reservoir computers are powerful tools for chaotic time series prediction. They can be trained to approximate phase space flows and can thus both predict future values to a high accuracy and reconstruct the general properties of a chaotic attractor without requiring a model. In this work, we show that the ability to learn the dynamics of a complex system can be extended to systems with multiple co-existing attractors, here a four-dimensional extension of the well-known Lorenz chaotic system. We demonstrate that a reservoir computer can infer entirely unexplored parts of the phase space; a properly trained reservoir computer can predict the existence of attractors that were never approached during training and, therefore, are labeled as unseen. We provide examples where attractor inference is achieved after training solely on a single noisy trajectory.A.R. and I.F. acknowledge support by the Spanish State Research Agency (AEI) through the Severo Ochoa and Maria de Maeztu Program for Centers and Units of Excellence in R&D (Grant No. MDM-2017-0711)